Question 3. Data Analysis and Predictive Modeling on Census data¶
Dataset: adult_modified.csv Dataset description here
a. Preprocessing and data analysis:
Examine the data for missing values. In case of categorical attributes, remove instances with missing values. In the case of numeric attributes, impute and fill-in the missing values using the attribute mean.
In [1]:
# Import Modules
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import os
from sklearn import preprocessing
from sklearn import neighbors, tree, naive_bayes
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn import cross_validation
In [5]:
# Change working directory
os.chdir('/Users/lavinia/Google Drive/18winter-textbooks/CSC478/Assignment2')
In [6]:
# Load file via read_csv
adult_df = pd.read_csv("adult-modified.csv")
adult_df.head(5)
Out[6]:
Exploratory Data Analysis
Understand the dataset and its variables
In [7]:
# Size of the dataset
adult_df.shape
Out[7]:
In [8]:
# Summarize the variables
adult_df.describe(include='all')
Out[8]:
In [9]:
# Find missing values in the dataset
adult_df.isnull().sum()
Out[9]:
Look at the csv file then we find out that missing valus are marked w/ '?'. Locate the columns with missing values.
In [10]:
adult_df.isin(['?']).any()
Out[10]:
In [11]:
# Change missing numeric values to column mean
adult_df.age[adult_df['age'] == '?'] = int(adult_df.age[adult_df['age'] != '?'].astype(int).mean())
#adult_df.head(20)
In [12]:
# Replace the '?' with NA in the categorical columns
adult_dfNew = adult_df.replace('?', np.NaN)
# Drop the NAs
adult_dfNew = adult_dfNew.dropna()
adult_dfNew.shape
Out[12]:
Question: Examine the characteristics of the attributes, including relevant statistics for each attribute, histograms illustrating the distribtions of numeric attributes, bar graphs showing value counts for categorical attributes, etc.¶
In [13]:
adult_dfNew.dtypes
Out[13]:
In [14]:
adult_dfNew.age = adult_dfNew.age.apply(int)
In [15]:
adult_dfNew.describe(include='all')
Out[15]:
In [16]:
# Histogram showing the distribution of age
adult_dfNew['age'].hist(grid = False)
Out[16]:
In [17]:
# Histogram showing the distribution of education
adult_dfNew['education'].hist(grid = False)
Out[17]:
In [18]:
# Histogram showing the distribution of hours-per-week
adult_dfNew['hours-per-week'].hist(grid = False)
Out[18]:
In [19]:
fig = plt.figure(figsize=(10,10))
fig.subplots_adjust(hspace=.5)
ax1 = fig.add_subplot(221)
ax1.set_xlabel('Workclass')
ax1.set_ylabel('Count')
ax1.set_title("Workclass Distribution")
adult_dfNew['workclass'].value_counts().plot(kind = 'bar', grid = False)
ax1 = fig.add_subplot(222)
ax1.set_xlabel('Marital Status')
ax1.set_ylabel('Count')
ax1.set_title("Martial Status Distribution")
adult_dfNew['marital-status'].value_counts().plot(kind = 'bar', grid = False)
ax1 = fig.add_subplot(223)
ax1.set_xlabel('Race')
ax1.set_ylabel('Count')
ax1.set_title("Race Distribution")
adult_dfNew['race'].value_counts().plot(kind='bar', grid = False)
ax1 = fig.add_subplot(224)
ax1.set_xlabel('Sex')
ax1.set_ylabel('Count')
ax1.set_title("Sex Distribution")
adult_dfNew['sex'].value_counts().plot(kind='bar', grid = False)
fig2 = plt.figure(figsize=(10,10))
fig2.subplots_adjust(hspace=.5)
ax1 = fig2.add_subplot(221)
ax1.set_xlabel('Income')
ax1.set_ylabel('Count')
ax1.set_title("Income Distribution")
adult_dfNew['income'].value_counts().plot(kind='bar', grid = False)
Out[19]:
Question: Perform the following cross-tabulations (including generating bar charts): education+race, workclass+income, work-class+race, and race+income. In the latter case (race+income) also create a table or chart showing percentages of each race category that fall in the low-income group. Discuss your observations from this analysis.¶
In [20]:
def generate_crosstab(first, second):
cross_tab = pd.crosstab(adult_dfNew[first], adult_dfNew[second])
plt.show(cross_tab.plot(kind="bar"))
generate_crosstab('education', 'race')
generate_crosstab('workclass', 'income')
generate_crosstab('workclass', 'race')
generate_crosstab('race', 'income')
In [21]:
# Create a table or chart showing percentages of each race category that fall in the low-income group.
pd.crosstab(adult_dfNew.race, adult_dfNew.income).apply(lambda x: x/x.sum(), axis = 1)
Out[21]:
So for each of the race we can see that most of the records n this data set are under low-income category. If we talk about each race, for the race Amer-Indian 90% of them are under low-income and around 10% are under high-income category, 77% of Asian are under low income category while 23% of them are under high-income category. For Black 87% are under low-income group and 13% are under high-income. For Hispanic 92% have income less than or equal to 50k so they are under low-income group while only 8% are in high-income category. For White 73% are under low-income category and 27% are under high-income category. Among all the races, in terms of race, highest percentage of people in low-income category are Hispanic and highest percentage of people under high-income category are White.
Question: Compare and contrast the characteristics of the low-income and high-income categories across the different attributes.¶
In [22]:
# Workclass vs income
pd.crosstab(adult_dfNew.workclass, adult_dfNew.income).apply(lambda x: x/x.sum(), axis = 1)
Out[22]:
In [23]:
# Workclass vs income
pd.crosstab(adult_dfNew.workclass, adult_dfNew.income).apply(lambda x: x/x.sum(), axis = 1)
Out[23]:
In [24]:
# Marital vs income
pd.crosstab(adult_dfNew['marital-status'], adult_dfNew.income).apply(lambda x: x/x.sum(), axis = 1)
Out[24]:
In [25]:
# Sex vs income
pd.crosstab(adult_dfNew.sex, adult_dfNew.income).apply(lambda x: x/x.sum(), axis = 1)
Out[25]:
From the analysis above, we see that members of the private class are most likely to make less than 50k, whereas those who are self employed are more likely to make over 50k. Even more notably, single people are signifcantly more likely to make less than 50k (93.33%) than married people (57.63%). Finally, 30% of males make more than 50k where only 11.89% of females make more than 50k.
b. Predictive Modeling and Model Evaluation:
Using either Pandas or Scikit-learn, create dummy variables for the categorical attributes. Then separate the target attribute ("income>50K") from the attributes used for training. [Note: you need to drop "income<=50K" which is also created as a dummy variable in earlier steps).]
In [26]:
# Create dummy variables for the categorical attributes. Then separate the target attribute ("income>50K") from the attributes used for training.
adult_dfNew = pd.get_dummies(adult_dfNew)
adult_dfNew.head(5)
Out[26]:
In [27]:
# Drop income <=50K
adult_dfNew = adult_dfNew.drop('income_<=50K', 1)
adult_dfNew.head(5)
Out[27]:
In [28]:
# Seperate target attribute ("income>50K") from the attributes used for training.
y = adult_dfNew['income_>50K']
y.head()
Out[28]:
In [29]:
# Remove target from rest of set
adult_dfNew = adult_dfNew.drop('income_>50K', 1)
adult_dfNew.head()
Out[29]:
Question: Use scikit-learn to build classifiers uisng Naive Bayes (Gaussian), decision tree (using "entropy" as selection criteria), and linear discriminant analysis (LDA). For each of these perform 10-fold crossvalidation (using cross-validation module in scikit-learn) and report the overall average accuracy.¶
In [30]:
# (Gaussian) naive Bayes classifier
nbclf = naive_bayes.GaussianNB()
nbclf = nbclf.fit(adult_dfNew, y)
cv_scores = cross_validation.cross_val_score(nbclf, adult_dfNew, y, cv = 10)
cv_scores
print("Overall Accuracy: %0.2f (+/- %0.2f)" % (cv_scores.mean(), cv_scores.std() * 2))
In [31]:
# Decision tree classifier
treeclf = tree.DecisionTreeClassifier(criterion = 'entropy', random_state = 9)
treeclf = treeclf.fit(adult_dfNew, y)
cv_scores = cross_validation.cross_val_score(treeclf, adult_dfNew, y, cv = 10)
cv_scores
print("Overall Accuracy: %0.2f (+/- %0.2f)" % (cv_scores.mean(), cv_scores.std() * 2))
In [32]:
# Linear Discriminant Analysis(LDA)
ldclf = LinearDiscriminantAnalysis()
ldclf = ldclf.fit(adult_dfNew, y)
cv_scores = cross_validation.cross_val_score(ldclf, adult_dfNew, y, cv = 10)
cv_scores
print("Overall Accuracy: %0.2f (+/- %0.2f)" % (cv_scores.mean(), cv_scores.std() * 2))
Question: For the decision tree model (generated on the full training data), generate a visualization of tree and submit it as a separate file (png, jpg, or pdf) or embed it in the Jupyter Notebook.¶
In [33]:
from sklearn.tree import export_graphviz
export_graphviz(treeclf,out_file='tree.dot', feature_names = adult_dfNew.columns)
In [35]:
import graphviz
with open("tree.dot") as f:
dot_graph = f.read()
graphviz.Source(dot_graph)
Out[35]:
In [36]:
system(dot -Tpng tree.dot -o dtree.png)
Out[36]:
In [37]:
from IPython.display import Image
Image(filename='dtree.png', width=800)
Out[37]:
